Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup CI action to automate testing #103

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ChAoSUnItY
Copy link

@ChAoSUnItY ChAoSUnItY commented Nov 23, 2024

Relates to #101, this PR adds a CI action file for testing test suites and self-hosting integrity when user pushes commits or opens a pull request, to prevent source code being incorrect on committed.

I haven't added an action for code style checking since there's no code style formatting guideline in the repository.

.github/workflows/main.yml Outdated Show resolved Hide resolved
.github/workflows/main.yml Outdated Show resolved Hide resolved
.github/workflows/main.yml Outdated Show resolved Hide resolved
.github/workflows/main.yml Outdated Show resolved Hide resolved
@jserv

This comment was marked as resolved.

@ChAoSUnItY ChAoSUnItY changed the title Add CI action for testing test suites and self-hosting on push and pull request Setup CI action to automate testing Nov 23, 2024
.github/workflows/main.yml Outdated Show resolved Hide resolved
.github/workflows/main.yml Outdated Show resolved Hide resolved
jserv

This comment was marked as resolved.

@fuhsnn
Copy link
Owner

fuhsnn commented Nov 24, 2024

It's a good chance to also include building these projects as regression testing https://github.com/fuhsnn/widcc/?tab=readme-ov-file#building-real-world-projects

And possibly with ASAN on, I'll test locally first.
I don't have a style guide, just intuitively "looks like chibicc's style", I think I can dig up an old clang-format file though.

I'm new to Github actions, but this action seems to worth using, any pro and cons of caching in your experience?
awalsh128/cache-apt-pkgs-action

@ChAoSUnItY
Copy link
Author

I haven't done any package caching in actions before, but current speed might be fast enough as long as matrix count is low and we reuse instances as much as possible.

Unless there are some critical scenarios where jobs must be separate out.

@fuhsnn
Copy link
Owner

fuhsnn commented Nov 24, 2024

I'm leaning towards running Dockerfiles with Github actions, I like having a more controllable environment and the ability to easily test multi-arch builds with https://github.com/docker/setup-qemu-action is nice.

@jserv
Copy link
Contributor

jserv commented Nov 25, 2024

It's a good chance to also include building these projects as regression testing https://github.com/fuhsnn/widcc/?tab=readme-ov-file#building-real-world-projects

Running SQLite and curl through automated GitHub Actions workflows makes a lot of sense. This approach helps catch both regressions and compatibility issues. We can also add continuous performance monitoring, similar to rv32emu's approach. The benchmark results automatically appear as comments in pull requests, making it easy for both contributors and reviewers to see how code changes impact performance compared to the default branch.

Another example: automated tests of ThreadX

@ChAoSUnItY
Copy link
Author

I am using nektos/act to test GitHub Actions locally, unfortunately, while it's all passed on GitHub Actions, on local Docker it will fail on test 1542 (after installing impacket, test 1451 also fail).

Currently I am looking on curl's official workflow to see if there's any hint on these 2 failures.

@fuhsnn
Copy link
Owner

fuhsnn commented Nov 25, 2024

I am using nektos/act to test GitHub Actions locally, unfortunately, while it's all passed on GitHub Actions, on local Docker it will fail on test 1542 (after installing impacket, test 1451 also fail).

Currently I am looking on curl's official workflow to see if there's any hint on these 2 failures.

That's why I prefer Docker for running the builds, you see, these code bases are decades old, some tests expect old features or behaviors from system tools that are deprecated or slightly changed over updates; a test failing don't always mean that the compiler is faulty. With Docker I can control exactly which dependency to install at what version on a per-project basis, which makes sense for regression testing (that is, what previously worked should keep working in the same non-changing environment).

I'm working on Docker based test jobs, just leave it to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants